Skip to content

Conversation

@VinInn
Copy link

@VinInn VinInn commented Jul 21, 2020

This PR improves the error estimation in CPEFast used on GPU providing a simple and fast parameterization in terms of charge, position in x and size in y.

The result is very close to what CPEGeneric provides w/o track-angle (actually a bit more accurate thanks to the estimation of the y-error using the cluster size in y (itself improved with information from the charge unbalance).

The main effect is a reduction and flattening of the chi2 and a reduction of the tails of the pulls.
Given this improvements I have (for the the being) simplified the cut in chi2 to a single value.
As side effect there is a small improvement in efficiency at least for single muon.

Single muon (2018 ideal): quadruplets-only and quadruplets+triplets
http://innocent.home.cern.ch/innocent/RelVal/SingleMuFlatIdeal_gpuNewErr25/plots_pixel_Pt09_pixel/effandfakePtEtaPhi.pdf
http://innocent.home.cern.ch/innocent/RelVal/SingleMuFlatIdeal_gpuNewErr25/plots_pixel_Pt09_pixel/tuning.pdf
http://innocent.home.cern.ch/innocent/RelVal/SingleMuFlatIdeal_gpuNewErr25/plots_pixel_Pt09_pixel/pulls.pdf
http://innocent.home.cern.ch/innocent/RelVal/SingleMuFlatIdeal_gpuNewErr25/plots_pixel_Pt09_pixel/resolutionsPt.pdf

ttBar PU50 2021 "realistic" (ape=3um!) (here with chi2 cut at 25 and 16)
http://innocent.home.cern.ch/innocent/RelVal/pixOnly2021_gpuNewErr25I/plots_pixel_Pt09_pixel/effandfakePtEtaPhi.pdf
http://innocent.home.cern.ch/innocent/RelVal/pixOnly2021_gpuNewErr25I/plots_pixelFromPVAllTP_pixel/effandfakePtEtaPhi.pdf
http://innocent.home.cern.ch/innocent/RelVal/pixOnly2021_gpuNewErr25I/plots_pixel_Pt09_pixel/tuning.pdf
http://innocent.home.cern.ch/innocent/RelVal/pixOnly2021_gpuNewErr25I/plots_pixel_Pt09_pixel/pulls.pdf
http://innocent.home.cern.ch/innocent/RelVal/pixOnly2021_gpuNewErr25I/plots_pixel_Pt09_pixel/resolutionsPt.pdf

p.s.
I take the opportunity to apologize with the authors and implementors of the fits: clearly the shape of chi2 and pulls seems to depend more on cluster properties and not on the fit itself (as I wrongly assumed for a year or so)

@VinInn VinInn requested a review from rovere July 21, 2020 12:09
@VinInn VinInn added enhancement Pixels Pixels-related developments labels Jul 21, 2020
@AdrianoDee
Copy link

AdrianoDee commented Jul 21, 2020

🚧 Validation running at fu-c2a02-35-02:/data/user/adiflori/patatrack-validation/run_521.Rpf85WLoVJ ...

trackQualityCuts.add<double>("chi2Scale", 25.)
->setComment(
"Factor to multiply the pT-dependent chi2 cut (currently: 30 for the broken line fit, 45 for the Riemann "
"Factor to multiply the pT-dependent chi2 cut (currently: 16 for the broken line fit, ?? for the Riemann "
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why the comment says 16 while the default is set to 25. ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. because I forgot to change it.
  2. after inspection of MTV we may decide that 16 is better than 25.

float sx[3], sy[3]; // the errors...
// float apeX, apexY; // ape^2
uint8_t sx2, sy1, sy2;
uint8_t sigmax[16], sigmax1[16], sigmay[16]; // in micron
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is 16 here ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the number of "bins"

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it obvious to people familiar with the CPE why 16 ?
I clearly have no idea :-)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely not. It is just my decision to use 16 bins. I use GenError as a black box.

@AdrianoDee
Copy link

Validation summary

Reference release CMSSW_11_1_0 at b7ad279
Development branch cms-patatrack/CMSSW_11_1_X_Patatrack at 935c1bb
Testing PRs:

Validation plots

/RelValTTbar_14TeV/CMSSW_11_1_0_pre8-PU_111X_mcRun3_2021_realistic_v4-v1/GEN-SIM-DIGI-RAW

  • tracking validation plots and summary for workflow 11634.5
  • tracking validation plots and summary for workflow 11634.501
  • tracking validation plots and summary for workflow 11634.502

/RelValZMM_14/CMSSW_11_1_0_pre8-111X_mcRun3_2021_realistic_v4-v1/GEN-SIM-DIGI-RAW

  • tracking validation plots and summary for workflow 11634.5
  • tracking validation plots and summary for workflow 11634.501
  • tracking validation plots and summary for workflow 11634.502

/RelValZEE_14/CMSSW_11_1_0_pre8-111X_mcRun3_2021_realistic_v4-v1/GEN-SIM-DIGI-RAW

  • tracking validation plots and summary for workflow 11634.5
  • tracking validation plots and summary for workflow 11634.501
  • tracking validation plots and summary for workflow 11634.502

Throughput plots

/EphemeralHLTPhysics1/Run2018D-v1/RAW run=323775 lumi=53

scan-136.885502.png
zoom-136.885502.png
scan-136.885522.png
zoom-136.885522.png

logs and nvprof/nvvp profiles

/RelValTTbar_14TeV/CMSSW_11_1_0_pre8-PU_111X_mcRun3_2021_realistic_v4-v1/GEN-SIM-DIGI-RAW

  • reference release, workflow 11634.5
  • development release, workflow 11634.5
  • development release, workflow 11634.501
  • development release, workflow 11634.502
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
  • development release, workflow 11634.511
  • development release, workflow 11634.512
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • cuda-memcheck --tool synccheck (report, log) found no CUDA-MEMCHECK results
  • development release, workflow 11634.521
  • development release, workflow 11634.522
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • cuda-memcheck --tool initcheck (report, log) found 939576 errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • cuda-memcheck --tool synccheck (report, log) found no CUDA-MEMCHECK results
  • development release, workflow 136.885502
  • development release, workflow 136.885512
  • development release, workflow 136.885522
  • testing release, workflow 11634.5
  • testing release, workflow 11634.501
  • testing release, workflow 11634.502
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
  • testing release, workflow 11634.511
  • testing release, workflow 11634.512
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • cuda-memcheck --tool synccheck (report, log) found no CUDA-MEMCHECK results
  • testing release, workflow 11634.521
  • testing release, workflow 11634.522
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • cuda-memcheck --tool initcheck (report, log) found 917248 errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • cuda-memcheck --tool synccheck (report, log) found no CUDA-MEMCHECK results
  • testing release, workflow 136.885502
  • testing release, workflow 136.885512
  • testing release, workflow 136.885522

/RelValZMM_14/CMSSW_11_1_0_pre8-111X_mcRun3_2021_realistic_v4-v1/GEN-SIM-DIGI-RAW

  • reference release, workflow 11634.5
  • development release, workflow 11634.5
  • development release, workflow 11634.501
  • development release, workflow 11634.502
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
  • development release, workflow 11634.511
  • development release, workflow 11634.512
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • cuda-memcheck --tool synccheck (report, log) found no CUDA-MEMCHECK results
  • development release, workflow 11634.521
  • development release, workflow 11634.522
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • cuda-memcheck --tool initcheck (report, log) found 46040 errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
  • development release, workflow 136.885502
  • development release, workflow 136.885512
  • development release, workflow 136.885522
  • testing release, workflow 11634.5
  • testing release, workflow 11634.501
  • testing release, workflow 11634.502
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
  • testing release, workflow 11634.511
  • testing release, workflow 11634.512
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • cuda-memcheck --tool synccheck (report, log) found no CUDA-MEMCHECK results
  • testing release, workflow 11634.521
  • testing release, workflow 11634.522
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • cuda-memcheck --tool initcheck (report, log) found 47000 errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
  • testing release, workflow 136.885502
  • testing release, workflow 136.885512
  • testing release, workflow 136.885522

/RelValZEE_14/CMSSW_11_1_0_pre8-111X_mcRun3_2021_realistic_v4-v1/GEN-SIM-DIGI-RAW

  • reference release, workflow 11634.5
  • development release, workflow 11634.5
  • development release, workflow 11634.501
  • development release, workflow 11634.502
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
  • development release, workflow 11634.511
  • development release, workflow 11634.512
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • cuda-memcheck --tool synccheck (report, log) found no CUDA-MEMCHECK results
  • development release, workflow 11634.521
  • development release, workflow 11634.522
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • cuda-memcheck --tool initcheck (report, log) found 26760 errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
  • development release, workflow 136.885502
  • development release, workflow 136.885512
  • development release, workflow 136.885522
  • testing release, workflow 11634.5
  • testing release, workflow 11634.501
  • testing release, workflow 11634.502
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
  • testing release, workflow 11634.511
  • testing release, workflow 11634.512
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • cuda-memcheck --tool synccheck (report, log) found no CUDA-MEMCHECK results
  • testing release, workflow 11634.521
  • testing release, workflow 11634.522
    • ✔️ step3.py: log
    • ✔️ profile.py: log
    • cuda-memcheck --tool initcheck (report, log) found 26544 errors
    • ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
    • ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
  • testing release, workflow 136.885502
  • testing release, workflow 136.885512
  • testing release, workflow 136.885522

Logs

The full log is available at https://patatrack.web.cern.ch/patatrack/validation/pulls/c69d40a2a8b44d51172c58eca8859024560dfc00/log .

@fwyzard
Copy link

fwyzard commented Jul 23, 2020

@VinInn looking at the validation plots for TTbar (e.g. here) there seems to be a small increase in the fake rate, while the pulls (e.g. here) seem unaffected.

Is it in line with your expectations ?

@VinInn
Copy link
Author

VinInn commented Jul 23, 2020

the small increase in the global fake rate is expected due to the simpler chi2 cut.
The fake rate for track associated to PV is negligibly affected.
Fake rate can always be reduced at expense of efficiency and of course optimized using fancy quality-estimators.

  1. Such estimators can be always applied later
  2. Their "robustness' has been often been questioned (see the DNN MVA saga)
    There is also the question if fit-quality shall be tuned using single tracks or in PU events as it seems that at large PU clusters are affected by some form of merging.

The chi2 is reduced and the tails of the pulls are better (comparison is orange vs black). statistics is low. The fact that average chi2 is dominated by "tails" does surprise me as well. And those tails seem to be dominated by deltarays (large charge) and "wrong" error for single pixels (the latter not fixed, maybe not even ok in original GenError).
I would like to stress that we do NOT have any fit result (of any type) of pixel-tracks using generic or template

@VinInn
Copy link
Author

VinInn commented Jul 28, 2020

sorry, but all wf are 11634 ?

@fwyzard
Copy link

fwyzard commented Jul 28, 2020

11634 is used to generate the workflows, but in fact we run only step3 and step4 over different samples

@fwyzard
Copy link

fwyzard commented Jul 28, 2020

@VinInn could you rebase this PR on top of CMSSW_11_2_X_Patatrack, and change the target branch accordingly ?

@VinInn
Copy link
Author

VinInn commented Jul 28, 2020

will do. still I prefer to keep a working version on 11_1

@VinInn
Copy link
Author

VinInn commented Aug 2, 2020

try to merge with 11_2 got
CONFLICT (add/add): Merge conflict in RecoTracker/TkSeedGenerator/plugins/SeedGeneratorFromL1TTracksEDProducer.cc
CONFLICT (add/add): Merge conflict in L1Trigger/TrackerDTC/interface/SetupRcd.h
CONFLICT (add/add): Merge conflict in L1Trigger/Phase2L1ParticleFlow/src/CaloClusterer.cc
CONFLICT (add/add): Merge conflict in L1Trigger/Phase2L1ParticleFlow/python/pfClustersFromHGC3DClusters_cfi.py
CONFLICT (add/add): Merge conflict in L1Trigger/Phase2L1ParticleFlow/plugins/PFClusterProducerFromHGC3DClusters.cc
CONFLICT (add/add): Merge conflict in L1Trigger/Phase2L1ParticleFlow/plugins/L1TPFProducer.cc
CONFLICT (add/add): Merge conflict in L1Trigger/Phase2L1ParticleFlow/plugins/L1TPFCaloProducer.cc
CONFLICT (content): Merge conflict in L1Trigger/L1THGCal/python/hgcalTriggerPrimitives_cff.py
CONFLICT (content): Merge conflict in L1Trigger/L1THGCal/python/egammaIdentification.py
CONFLICT (add/add): Merge conflict in L1Trigger/DTTriggerPhase2/plugins/DTTrigPhase2Prod.cc
CONFLICT (content): Merge conflict in L1Trigger/Configuration/python/SimL1Emulator_cff.py
CONFLICT (content): Merge conflict in L1Trigger/Configuration/python/L1Trigger_EventContent_cff.py
CONFLICT (content): Merge conflict in L1Trigger/CSCTriggerPrimitives/src/GEMCoPadProcessor.cc
CONFLICT (content): Merge conflict in L1Trigger/CSCTriggerPrimitives/src/CSCGEMMotherboard.cc
CONFLICT (content): Merge conflict in HLTrigger/Configuration/test/OnLine_HLT_PRef.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/test/OnLine_HLT_PIon.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/test/OnLine_HLT_HIon.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/test/OnLine_HLT_GRun.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/test/OnLine_HLT_Fake2.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/test/OnLine_HLT_Fake1.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/test/OnLine_HLT_Fake.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/test/OnLine_HLT_FULL.py
CONFLICT (modify/delete): HLTrigger/Configuration/test/OnLine_HLT_2018.py deleted in HEAD and modified in VinInn/NewFastErr. Version VinInn/NewFastErr of HLTrigger/Configuration/test/OnLine_HLT_2018.py left in tree.
CONFLICT (content): Merge conflict in HLTrigger/Configuration/python/HLT_PRef_cff.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/python/HLT_PIon_cff.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/python/HLT_HIon_cff.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/python/HLT_GRun_cff.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/python/HLT_Fake_cff.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/python/HLT_Fake2_cff.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/python/HLT_Fake1_cff.py
CONFLICT (content): Merge conflict in HLTrigger/Configuration/python/HLT_FULL_cff.py
CONFLICT (modify/delete): HLTrigger/Configuration/python/HLT_2018_cff.py deleted in HEAD and modified in VinInn/NewFastErr. Version VinInn/NewFastErr of HLTrigger/Configuration/python/HLT_2018_cff.py left in tree.
CONFLICT (content): Merge conflict in Geometry/HGCalGeometry/src/HGCalGeometry.cc
CONFLICT (content): Merge conflict in Geometry/HGCalCommonData/src/HGCalGeomParameters.cc
CONFLICT (content): Merge conflict in Geometry/HGCalCommonData/src/HGCalDDDConstants.cc
CONFLICT (add/add): Merge conflict in Geometry/HGCalCommonData/interface/HGCalTypes.h
CONFLICT (content): Merge conflict in Geometry/HGCalCommonData/data/dd4hep/testHGCalV12.xml
CONFLICT (content): Merge conflict in Geometry/CaloTopology/src/HGCalTopology.cc
CONFLICT (content): Merge conflict in Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2026D51.xml
CONFLICT (content): Merge conflict in Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2026D50.xml
CONFLICT (content): Merge conflict in Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2026D49.xml
CONFLICT (content): Merge conflict in DQM/Integration/python/clients/sistrip_dqm_sourceclient-live_cfg.py
CONFLICT (content): Merge conflict in DQM/BeamMonitor/plugins/BeamMonitor.h
CONFLICT (content): Merge conflict in Configuration/StandardSequences/python/L1TrackTrigger_cff.py
CONFLICT (content): Merge conflict in Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py
CONFLICT (content): Merge conflict in Configuration/PyReleaseValidation/python/relval_2026.py
CONFLICT (content): Merge conflict in Configuration/Geometry/python/dict2026Geometry.py
CONFLICT (modify/delete): Configuration/Eras/python/Era_Phase2C8_cff.py deleted in HEAD and modified in VinInn/NewFastErr. Version VinInn/NewFastErr of Configuration/Eras/python/Era_Phase2C8_cff.py left in tree.
CONFLICT (content): Merge conflict in Configuration/AlCa/python/autoCond.py

@fwyzard
Copy link

fwyzard commented Aug 2, 2020

Indeed... that's because the history of the two branches has diverged right before CMSSW_11_1_0_Patatrack, and you are seeing the conflicts between CMSSW_11_1_0 and CMSSW_11_2_0_pre2.

Rather than merging into 11_2_X_Patatrack or master, it should be easier to cherry-pick the commits:

git checkout CMSSW_11_2_X_Patatrack
git cherry-pick 2ac98ff d9c676b 1d574c7 55f87f8

or, to work with the nightly:

git checkout master
git cherry-pick 2ac98ff d9c676b 1d574c7 55f87f8

@VinInn
Copy link
Author

VinInn commented Aug 3, 2020

superseded by #528

@VinInn VinInn closed this Aug 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Pixels Pixels-related developments

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants